Methods and systems for transmitting packets through aggregated end-to-end connection

ABSTRACT

A method and a first communications router for transmitting data packets to a second communications router by establishing an aggregated end-to-end connection with the second communications router. The aggregated end-to-end connection comprises a plurality of established end-to-end connections. Data packets are transmitted through a first established end-to-end connection when a first condition is satisfied, and through a second established end-to-end connection when a second condition is satisfied. The first and second established end-to-end connections belong to the aggregated end-to-end connection. The first communications router comprises a plurality of network interfaces.

RELATED APPLICATIONS

The present application is a non-provisional continuation applicationwhich claims the benefits of and is based on U.S. application Ser. No.15/726,318 titled “METHODS AND SYSTEMS FOR TRANSMITTING PACKETS THROUGHAGGREGATED END-TO-END CONNECTION” filed on Oct. 5, 2017, which is acontinuation application which claims the benefits of and is based onU.S. application Ser. No. 13/935,578 titled “METHODS AND SYSTEMS FORTRANSMITTING PACKETS THROUGH AGGREGATED END-TO-END CONNECTION” filed onJul. 5, 2013, now U.S. Pat. No. 9,787,501, which is acontinuation-in-part application which further claims the benefits ofand is based on U.S. application Ser. No. 12/646,774 titled “THROUGHPUTOPTIMIZATION FOR BONDED VARIABLE BANDWIDTH CONNECTIONS” filed on Dec.23, 2009, now U.S. Pat. No. 9,019,827, the disclosures of which arehereby incorporated by specific reference thereto.

TECHNICAL FIELD

The present invention relates in general to the field of computernetworks. More particularly, the present invention relates to methodsand systems for establishing an aggregated end-to-end connection with aplurality of established end-to-end connections between a first andsecond communication router. The present invention further relates tomethods carried out by the first communications router for transmittingdata packets to the second communications router through specificestablished end-to-end connections based on whether certain conditionsare satisfied or not.

BACKGROUND ART

A multi Wide Area Network (WAN) Site-to-Site VPN router is a router thatsupports aggregating the bandwidth of multiple interconnections, e.g.,WAN connections for accessing one or more remote private networks. Insome implementations, each TCP/IP session is routed to only one WAN. Inthis configuration, a single TCP file transfer session can only utilizethe bandwidth of one WAN connection on each end. For example, in asession based site-to-site virtual private network (VPN) connection VPNtraffic is routed to multiple WAN connections between two sites (e.g.,sites A and B).

In one implementation, M×N tunnels are initially formed between the WANconnections where M and N are the number of WAN network connections ofsite A and site B, respectively. Application TCP/IP sessions are thenrouted over the different tunnels. It is notable, however, that while asession based site-to-site VPN is able to utilize different tunnels fordifferent sessions, a single download session in this type of connectionis only able to utilize one tunnel.

In order to increase throughput of single data transfer sessions,routing schemes have been created in an attempt to utilize multiple WANconnections for a single TCP/IP session. In some cases channel bondingis implemented. For example, in bonded site-to-site VPN connections onmulti WAN routers, data from a single TCP/IP session is distributed tomultiple tunnels. In these arrangements, the M×N tunnels are utilizedaccording to their respective uplink or downlink bandwidth, and theamount of data in a single session that is sent over these tunnels isdistributed in proportion to the individual bandwidth ratios of thetunnels. These solutions, however, still fail to take into the accountthat some tunnels should not be used for transmitting data because ofone or more conditions. Furthermore, these solutions also do not takeinto account that some network interfaces fail to satisfy one or moreconditions, and thus those network interfaces should not be used fortransmitting data.

DISCLOSURE OF THE INVENTION Summary of Invention

Accordingly, the present disclosure provides for devices, systems, andmethods which select established end-to-end connections by classifyingestablished end-to-end connections into a first group and at least onenon-first group of established end-to-end connection(s), wherein thefirst group of established end-to-end connection(s) satisfy all of oneor more conditions and wherein the at least one non-first group ofestablished end-to-end connection(s) does not satisfy all of the one ormore conditions. By using conditions, embodiments of the presentinvention are able to select established end-to-end connections thathave satisfied the conditions to transmit data packets while not usingnon-satisfactory established end-to-end connections to transmit datapackets.

According to one embodiment of the present invention, when a firstestablished end-to-end connection does not satisfy a first condition, asecond established end-to-end connection is used to transmit datapackets. According to one embodiment of the present invention, when thefirst established end-to-end connection satisfy a fourth condition, thefirst established end-to-end connection is used again to transmit datapackets and the second established end-to-end connection stopping beingused to transmit data packets.

According to one embodiment of the present invention, the fourthcondition is more stringent that the first condition to avoid togglingbetween satisfying the fourth conditions but fails the first conditionand results in switching back and forth of the first establishedend-to-end connection and the second established end-to-end connection.

According to one embodiment of the present invention, the secondestablished end-to-end connection is used if it has satisfied the secondcondition.

According to one embodiment of the present invention, before ceasing touse the first established end-to-end connection to transmit datapackets, a third condition is used to determine when to initialize,prepare or activate the second established end-to-end connection fortransmitting data packets when the first established end-to-endconnection fails the first condition.

DETAILS OF INVENTION

The ensuing description provides preferred exemplary embodiment(s) andexemplary embodiments only, and is not intended to limit the scope,applicability or configuration of the invention. Rather, the ensuingdescription of the preferred exemplary embodiment(s) and exemplaryembodiments will provide those skilled in the art with an enablingdescription for implementing a preferred exemplary embodiment of theinvention. It is understood that various changes may be made in thefunction and arrangement of elements without departing from the spiritand scope of the invention as set forth in the appended claims.

Also, it is noted that the embodiments may be described as a processwhich is depicted as a flowchart, a flow diagram, a data flow diagram, astructure diagram, or a block diagram. Although a flowchart may describethe operations as a sequential process, many of the operations can beperformed in parallel or concurrently. In addition, the order of theoperations may be re-arranged. A process is terminated when itsoperations are completed, but could have additional steps not includedin the figure. A process may correspond to a method, a function, aprocedure, a subroutine, a subprogram, etc. When a process correspondsto a function, its termination corresponds to a return of the functionto the calling function or the main function.

Moreover, as disclosed herein, the term “secondary storage” and “mainmemory” may represent one or more devices for storing data, includingread only memory (ROM), random access memory (RAM), magnetic RAM, corememory, magnetic disk storage mediums, optical storage mediums, flashmemory devices and/or other machine readable mediums for storinginformation. The term “machine readable medium” includes, but is notlimited to portable or fixed storage devices, optical storage devices,wireless channels and various other mediums capable of storing,containing or carrying instruction(s) and/or data. A machine-readablemedium can be realized by virtualization, and can be a virtual machinereadable medium including a virtual machine readable medium in acloud-based instance.

Furthermore, embodiments may be implemented by hardware, software,firmware, middleware, microcode, hardware description languages, or anycombination thereof. When implemented in software, firmware, middlewareor microcode, the program code or code segments to perform the necessarytasks may be stored in a machine readable medium such as storage medium.A processing unit(s) may perform the necessary tasks. A processingunit(s) can be a CPU, an ASIC semiconductor chip, a semi-conductor chip,a logical unit, a digital processor, an analog processor, a FPGA or anyprocessor that is capable of performing logical and arithmeticfunctions. A code segment may represent a procedure, a function, asubprogram, a program, a routine, a subroutine, a module, a softwarepackage, a class, or any combination of instructions, data structures,or program statements. A code segment may be coupled to another codesegment or a hardware circuit by passing and/or receiving information,data, arguments, parameters, or memory contents. Information, arguments,parameters, data, etc. may be passed, forwarded, or transmitted via anysuitable means including memory sharing, message passing, token passing,network transmission, etc. A processing unit(s) can be realized byvirtualization, and can be a virtual processing unit(s) including avirtual processing unit in a cloud-based instance.

An end-to-end connection can use connection-oriented protocol, such asTransmission Control Protocol (TCP), or connectionless protocol, such asUser Datagram Protocol (UDP), to transmit data packets. Well-knownprotocols for deploying end-to-end connections include Layer 2 TunnelingProtocol (L2TP), secure shell (SSH) protocol, Multi-protocol LabelSwitching (MPLS), and Microsoft's Point-to-Point Tunneling Protocol(PPTP). A connection connected to a network interface is in the form ofoptical fiber, Ethernet, ATM, Frame Relay, T1/E1. IPv4, IPv6, wirelesstechnologies, WI-FI, WiMax, High-Speed Packet Access technology, 3GPPLong Term Evolution (LTE) or the like.

A network interface can be a virtual network interface, including avirtual network interface in a cloud based instance.

FIG. 1A illustrates a network environment according to one of theembodiments of the present invention. Communications router 106establishes an aggregated end-to-end connection 103 with communicationsrouter 108. Aggregated end-to-end connection 103 comprises a pluralityof established end-to-end connections. The plurality of establishedend-to-end connections is classified into a plurality of groups. Forbetter illustration, group 103A represents a first group of establishedend-to-end connections, group 103B represents a second group ofestablished end-to-end connections, and group 103C represents all otherestablished end-to-end connections that do not belong to the first groupor the second group of established end-to-end connections.Classification of established end-to-end connections into group 103A,103B, or 103C is discussed in further detail herein. Alternatively, theplurality of established end-to-end connections is classified intomultiple groups.

Communications routers 106 and 108 have one or more network interfacesaccording to one of the embodiments. Communications router 106establishes a plurality of established end-to-end connections via one ormore of its network interfaces with one or more network interfaces ofcommunications router 108.

In one of the embodiments, after the classification of establishedend-to-end connections into groups 103A, 103B and 103C, data packets aretransmitted by communications router 106 via different groups ofestablished end-to-end connections according to conditions set by auser, manufacturer or administrator of communications router 106.

In one of the embodiments, after the classification of establishedend-to-end connections into groups 103A, 103B and/or 103C, differenttypes of packets are transmitted by communications router 106 viadifferent groups of established end-to-end connections according toconditions set by a user, manufacturer or administrator ofcommunications router 106 and type of packets.

Aggregated End-to-End Connection

A plurality of established end-to-end connections are aggregated,combined or bonded together to form one aggregated end-to-endconnection, such as aggregated end-to-end connection 103. Those skilledin the arts would appreciate that there are myriad ways to aggregate,combine, or bond a plurality of established end-to-end connections toform one aggregated end-to-end connection. An aggregated end-to-endconnection is perceived as one end-to-end connection by sessions orapplications that are using it. An aggregated end-to-end connection canbe perceived as a tunnel, a virtual private network or connection orconnectionless oriented connection. For example, aggregated end-to-endconnection 103 is a TCP connection. In another example, an aggregatedend-to-end connection 103 is a UDP connection. In another example,aggregated end-to-end connection is an aggregation of a plurality oftunnels, and each tunnel is linked between communications router 106 andcommunications router 108. In another example, aggregated end-to-endconnection 103 is a VPN tunnel, comprising a plurality of establishedend-to-end connections, and each established end-to-end connection islinked between communications router 106 and communications router 108.

Generally, communications router 106 encapsulates data packetsoriginating from a transmitting host into at least one data packet. Forreadability, the data packets originating from a transmitting host arereferred to as original data packets. The transmitting host can beconnected to communications router 106 through a cable, a wirelessconnection, a near-field connection transmission or through anothernetwork device. When communications router 108 receives the data packetsthat have encapsulated the original data packets, communications router108 decapsulates the original data packets from the data packets andthen forward the original data packets to a receiving host or node.

There are myriad ways to encapsulate original data packets in datapackets sent by communications router 106. In particular, communicationsrouter 106 may tunnel original data packets between its logicalendpoints using known layer-2 and/or layer-3 tunneling protocols, suchas the Layer-2 Tunneling Protocol (L2TP) and the Generic RoutingEncapsulation (GRE) protocol. In this case, one or more tunnel headersare prepended to a data packet to appropriately route the data packetalong the virtual circuit. The Multi-Protocol Label Switching (MPLS)protocol may be used as a tunneling mechanism for establishing layer-2virtual circuits or layer-3 network-based VPNs through an IP network.

Communications router 106 may use TCP, UDP or other communicationprotocols as the communication protocol for an established end-to-endconnection. One of the biggest challenges faced when using aggregatedend-to-end connection is to re-order data packets received amongestablished end-to-end connections because received data packets mayarrive at the communications router 108 out-of-order. If communicationsrouter 108 delivers the received data packets too early to the receivinghost, the receiving host may consider those data packets that have notbeen received as lost data packets even if those data packets eventuallyarrives. One way to overcome this challenge is when communicationsrouter 106 transmits data packets to communications router 108,communications router 106 assigns each data packet with a sequencenumber. Communications router 108 can then re-order the data packetsaccording to the sequence number.

Between communications router 106 and communications router 108, therecan be one or more aggregated end-to-end connections. Each aggregatedend-to-end connection has its own established end-to-end connections.For example, a first aggregated end-to-end connection may have tenestablished end-to-end connections while a second aggregated end-to-endconnection may have five established end-to-end connections. The firstaggregated end-to-end connection and the second aggregated end-to-endconnection cannot use the same established end-to-end connection.

FIG. 5 is an illustrative block diagram of a communications router, suchas communications router 106, according to one of the embodiments of thepresent invention. Communications router 106 comprises processing unit502, main memory 503, system bus 504, secondary storage 505, andplurality of network interfaces 506. Processing unit 502 and main memory503 are connected to each other directly. System bus 504 connectsprocessing unit 502 directly or indirectly to secondary storage 505, andplurality of network interfaces 506. Using system bus 504 allowscommunications router 106 to have increased modularity. System bus 504couples processing unit 502 to secondary storage 505, and plurality ofnetwork interfaces 506. System bus 504 can be any of several types ofbus structures including a memory bus, a peripheral bus, and a local bususing any of a variety of bus architectures. Secondary storage 505stores program instructions for execution by processing unit 502.Secondary storage 505 further stores conditions, wherein classificationof established end-to-end connections into different groups depends onwhether or not the established end-to-end connections satisfy theconditions.

FIG. 4 is a tree-diagram illustrating how established end-to-endconnections are classified into groups according to one of theembodiments. Established end-to-end connections 400 represents allestablished end-to-end connections between two network nodes, such ascommunications routers 106 and 108. Established end-to-end connectionsthat satisfy conditions to be in a first group are classified into thefirst group of established end-to-end connections 410. The establishedend-to-end connections that do not satisfy conditions to be classifiedinto first group of established end-to-end connections 410 areclassified to non-first group of established end-to-end connections 420.In one variant, non-first group of established end-to-end connections420 is further classified into second group of established end-to-endconnections 421 and non-first/second established end-to-end connections422. The established end-to-end connections that do not satisfyconditions to be classified to second group of established end-to-endconnections 421 are classified to non-first/second establishedend-to-end connections 422. In one variant, non-first group ofestablished end-to-end connections is not further classified into secondgroup of established end-to-end connections 421 and non-first/secondgroup of established end-to-end connections 422. Therefore second groupof established end-to-end connections 421 is omitted.

In order to classify established end-to-end connections into differentgroups, a communications router determines if the established end-to-endconnections satisfy certain conditions. According to one of theembodiments, the classification is performed in order to transmitcertain types of packets through established end-to-end connectionsbelonging to a certain group, and to transmit other types of packetsusing established end-to-end connections in other groups. According toone of the embodiments, once the classification is done, the groups areranked so that data packets with highest priority are transmitted usinga group of established end-to-end connections with highest rank.

In one of the embodiments of the present invention, when there are aplurality of established end-to-end connections in the first group ofestablished end-to-end connections, data packets are distributed amongand transmitted through the plurality of established end-to-endconnections the first group using load balancing technology, or anyother technology that allows data packets to be distributed andtransmitted among multiple established end-to-end connections. It isknown to those skilled in the arts that there are many techniques totransmit data packets using multiple established end-to-end connections.The same techniques can be applied to distribute data packets in othergroups of established end-to-end connections as well.

FIG. 1B illustrates system 100 adapted according to embodimentsconfigured to optimize the throughput of bonded multiple variablebandwidth connections by adjusting a tunnel bandwidth weighting schemaduring a data transfer session. System 100 includes multiple sites 102and 104, which each comprise at least one communications router 106 and108. Communications routers 106 and 108 may be embodied as multi WANrouters which support aggregating the bandwidth of multiple Internetconnections. Communications routers 106 and 108 are connected overnetwork 110. Network 110 may comprise a local area network (LAN),metropolitan area network (MAN), wide area network (WAN), wirelessnetwork, the public switched telephone network (PSTN), the Internet, anintranet, an extranet, etc.

Site 102 and router 106 may comprise M connections 112, and site 104 androuter 108 may comprise N connections 114. Connections 112 and 114 aredata connections for communicating information within network 110between sites 102 and 104. In the illustrated embodiment, M is equal to3 and N is equal to 2; however, these values may vary according todesired routers and configurations. Connections 112 and 114 may havesimilar or differing bandwidth capabilities. Further, connections 112and 114 may comprise different types of WAN connections, such as aWI-FI, cable, DSL, T1, 3G, 4G, satellite connections, and the like. Itis also noted that site 102 and site 104 may be thought of as both asender or receiver, and discussions regarding the functionality ofeither site may be implemented on the other site. In other words, system100 may be implemented as a symmetrical network.

FIG. 2A shows a high level flow diagram of operation of system 100depicting a method 200 for increasing throughput of a bonded connection.It should be appreciated that the particular functionality, the order ofthe functionality, etc. provided in FIG. 2A is intended to be exemplaryof operation in accordance with the concepts of the present invention.Accordingly, the concepts herein may be implemented in various waysdiffering from that of the illustrated embodiment.

At block 201 of the illustrated embodiment when establishing a bondedconnection between routers 102 and 104, such as by implementing a bondedsite-to-site VPN connection, M×N virtual tunnels 116 may be created.Virtual tunnels 116 correspond to a unique permutation of the networkconnections of site 102 and the network connections of site 104.

Aggregated end-to-end connection 103 is equivalent to M×N virtualtunnels 116 as routers 106 and 108 aggregate M×N virtual tunnels 116 toform one aggregated end-to-end connection 103. M×N virtual tunnels 116are classified into groups 103A, 103B and 103C.

At block 202 of the illustrated embodiment, default weights for thetunnels are determined and/or assigned. To determine default weightsembodiments exchange uplink and downlink bandwidth data of connections112 and 114 between sites 102 and 104. Using this bandwidth data, adefault weight may be calculated according to the following: supposesite 102's downlink bandwidths of connections 1 to m are d1, d2, . . .dm, and site 104's uplink bandwidths of connections 1 to n are u1, u1, .. . , un; the default weight for the tunnel between site 102'sconnection X and site 104's connection Y may be defined as DW(x,y),where DW(x,y)=dx·uy.

Using the above method to calculate default weight, if connections 112-1through 112-3 are WAN connections of a multi WAN router with respectiveuplink/downlink bandwidths of 10M/6M, 8M/4M, and 6M/6M, and connections114-1 through 114-2 are WAN connections of a multi WAN router withrespective uplink/downlink bandwidths of 7M/5M and 9M/3M, the respectivedefault weights for each tunnel will be as follows:

TABLE 0001 For site 102 For site 104 DW(1, 1) = 6 * 7 = 42 DW(1, 1) =5 * 10 = 50 DW(1, 2) = 6 * 9 = 54 DW(1, 2) = 5 * 8 = 40 DW(2, 1) = 4 * 7= 28 DW(1, 3) = 5 * 6 = 30 DW(2, 2) = 4 * 9 = 36 DW(2, 1) = 3 * 10 = 30DW(3, 1) = 6 * 7 = 42 DW(2, 2) = 3 * 8 = 24 DW(3, 2) = 6 * 9 = 54 DW(2,3) = 3 * 6 = 18

It is noted that other ways to calculate default weight arecontemplated, and the above is simply an example of the implementationof an embodiment of the present invention. It is noted that manydifferent weighting schema may be used to define the initial bandwidthof a tunnel. For example, one may desire to only weight a tunnel in onedirection using the downlink capacity of a receiving site and the uplinkcapacity of the sending site. Any weighting scheme used to characterizecapacity of the tunnels at the establishment of the bonded connectionmay be used for the purposes of the present invention.

When packets are being routed from site 102 to site 104 according toembodiments, the packets will be distributed to the tunnels in a ratioaccording to an effective weight, EW(x,y). Initially the effectiveweight of embodiments is set to be equal to the default weight,EW(x,y)=DW(x,y), and if the bandwidth of tunnels 116 remains unchangedfrom the initial setting, the effective weight is optimal for packetdistribution. However, if a user is downloading a file over a bondednetwork connection in a TCP session with one or more tunnels havingpacket drops, the overall throughput of the session will dropdramatically. This is in part because the packet drops will keep causingTCP retransmissions and TCP's flow control will maintain a lowerthroughput even though tunnels without packet drops are not fullyoccupied.

One effective way to increase throughput would be to avoid such packetdrops. To do so, embodiments of the present invention discern whentunnels are experiencing an increase or decrease in packet drop rates atblock 203 of the illustrated embodiment. Embodiments further function tomodify the effective weight of tunnels which are experiencing or haveexperienced changes in packet drop rates at block 204. The packet droprate information may be monitored continuously or be monitored based onspecific time periods. Once it is determined that a tunnel isexperiencing an unacceptable rate of packet drops (block 204-1), theillustrated embodiment decreases the effective weight of the tunnel atblock 204-2. In some embodiments, unacceptable may mean that the packetdrop rate is a non-zero quantity, while other embodiments may determinethat an unacceptable rate is any rate beyond a predefined threshold.Embodiments implement these decreases in stepwise fashion, in acontinuous manner, in a reduction at one time in proportion to theincrease in the packet drop rate, etc. When reductions are done in agradual manner, embodiments may continue to monitor the tunnel in orderto optimize the amount of reduction which is implemented.

Tunnels 116 may be established or monitored by sending heartbeat packetsthrough each tunnel from either router 106 or router 108. In someembodiments when the receive end fails to receive heartbeat packets froma tunnel for a period of time, it will treat that tunnel as down and thetunnel will not be used for routing traffic. If heartbeat packets againstart being received, the tunnel may be re-established and be weightedalong with the other tunnels. As such, in the event that all packets arebeing dropped in a tunnel and the effective weight of that tunnel isreduced to zero, embodiments may utilize heartbeat packets to monitorand re-establish a connection.

Moreover, when tunnels recover all or part of their respectivebandwidths, e.g. it is determined that the packet drop rate decreases(block 204-3), the illustrated embodiment functions to increase theeffective weight of such tunnels (block 204-4) in order to fully, ormore fully, utilize the bandwidth. Some embodiments increase theeffective weight for a tunnel using predetermined step sizes until anaccurate effective weight is regained. Other embodiments increase theeffective weight proportionate to a newly measured bandwidth which maycorrespond to a newly measured packet drop rate.

Moreover, embodiments may increase the effective weight for a tunnelbased on a predetermined linear or exponential scale.

After the effective weight of the tunnels are adjusted, or it isdetermined that no adjustment is needed, the weighting scheme of thesystem is updated at block 205 of the illustrated embodiment. Thisupdate may comprise storing any processed information, using suchinformation in further processing, causing the system to take no action,etc. For example, processing performed with respect to block 205 mayoperate to average weighting schemes over a period of time, such as tomitigate error associated with highly transient anomalies. Further, theupdated information may be used on system 100 to modify the packetdistribution of the data transfer session, as discussed with respect toFIG. 2B. System 100 may continue to implement steps 203-205 continuouslyor periodically throughout a data transfer session.

FIG. 2B illustrates an embodiment where, after weighting method 200 isimplemented, the packets are distributed based, at least in part, on themodified weight of the tunnels. Specifically, block 206 of theillustrated embodiment operates to distribute packets across the tunnelsin accordance with the weighting scheme determined by operation ofmethod 200. In some embodiments, this distribution will changethroughout a data transfer session, and therefore the steps of FIG. 2Bare shown as repeating. Some embodiments change the packet distributioneach time the system is updated at block 205. Moreover, block 205 maycause changes to be implemented periodically, in response to certaindrop rate change thresholds, etc. It should be appreciated that thedetermination of weighting by operation of method 200 and theapplication of determined weighting to packet distribution at block 206may have different periodicity. For example, method 200 may operate toprovide updates of weighting scheme information using a relatively shortiterative cycle while the distribution of packets is altered based uponsuch weighting scheme information using a longer iterative cycle.

To monitor the bandwidth of the various tunnels 116, some embodiments ofthe present invention encapsulate each transmitted IP packet withvarious information. FIG. 3 illustrates an example embodiment showingthe type of information 300 which may be encapsulated in a transmittedIP packet. Version field 302 may contain information about the protocolversion being utilized and protocol type field 303 may contain theprotocol type of the payload packet. In general, the value of this fieldwill correspond to the Ethernet protocol type for the packet. However,additional values may be defined in other documents. Tunnel ID field 304may be a 32-bit field and may contain an identifier to identify thecurrent tunnel of the IP packet. Advanced Encryption Standard (AES)initialization vector field 306 may be a 32-bit field and may contain aninitialization vector for AES encryption. Global sequence number field308 may be a 32-bit field and may contain a sequence number which isutilized to re-sequence each of the packets for various sessions intothe proper order when they have emerged from their respective tunnels.Per tunnel sequence number field 310 may be a 32-bit field which mayrepresent a sequence number that is assigned to each packet routed to aparticular tunnel. AES encrypted payload field 312 may be utilized toconvey the payload of the IP packet.

The per tunnel sequence number discussed above may be used to monitordropped packets in a tunnel. In one embodiment the router on thereceiving end calculates the packet drop rate of each tunnel, DR(x,y),every f seconds by monitoring the per tunnel sequence number of thereceived packets. DR(x,y) may be characterized as the sequence numbersmissed divided by a sequence number increase for a period f. The lengthof period f may vary, and in one embodiment f is equal to 5 seconds.

Other methods may also be used to monitor dropped packets, e.g.: thesender may periodically inform the receive end how many packets it hassent, the sender sends a heartbeat packet to the receive end everyconstant period of time and the receive end can estimate the overalldrop rate by monitoring the heartbeat packets' drop rate, by acquiringdrop rate figures from physical interface/device/layer, etc.

The receive end may feedback a particular tunnel's drop rate, effectiveweight, or other bandwidth indicators, to the sending router. When thesender receives information regarding packet drops, some embodimentslower the effective weight EW(x,y) of a tunnel by EW(x,y)·DR(x,y). Othermetrics may be used to modify the effective weight of a tunnel. In someembodiments, the sender may receive feedback and the effective weightmay be reduced by number that is greater than or less than the packetdrop rate. Such variances may be configured according to the particularneeds of a communication system. The above example represents a metricthat attempts to lower the effective weight of the tunnel to a weightwhich prevents further packet drops while maximizing the amount ofusable bandwidth of the tunnel. Any metric which finds this balance maybe preferred.

FIG. 6A is a flowchart illustrating a process used to classify anestablished end-to-end connection into one of the groups according toone of the embodiments of the present invention. FIG. 6A is viewed inconjunction with FIG. 5 for better understanding of the embodiment. Instep 601, processing unit 502 retrieves conditions from secondarystorage 505. The classification is based on whether or not theestablished end-to-end connection satisfies the conditions retrieved.

In step 602, processing unit 502 determines whether or not anestablished end-to-end connection between communications router 106 andcommunications router 108 satisfies the conditions.

If, in step 602, processing unit 502 determines that an establishedend-to-end connection satisfies all conditions, then the establishedend-to-end connection belongs to a first group of established end-to-endconnections in step 603.

If, in step 602, processing unit 502 determines that an establishedend-to-end connection satisfies at least one conditions, but not allconditions, then the established end-to-end connection belongs to asecond group of established end-to-end connections in step 604.

If, in step 602, processing unit 502 determines that an establishedend-to-end connection satisfies one or more specific conditions, but notall conditions, then the established end-to-end connection belongs to asecond group of established end-to-end connections in step 604.

In one variant, if, in step 602, processing unit 502 determines that anestablished end-to-end connection does not satisfy any condition(s),then the established end-to-end connection belongs to non-first/secondgroup of established end-to-end connections in step 605. Establishedend-to-end connections that belong to the non-first/second group do notbelong to the first group or the second group of established end-to-endconnections.

In one variant, there is no second group of established end-to-endconnections, and step 604 is omitted. If, in step 602, processing unit502 determines that an established end-to-end connection does notsatisfy all conditions, then the established end-to-end connectionbelongs to the non-first/second group in step 605.

In one variant, there is only one condition retrieved from secondarystorage 505 in step 601. If, in step 602, processing unit 502 determinesthat an established end-to-end connection satisfies the condition, thenthe established end-to-end connection belongs to a first group ofestablished end-to-end connections in step 603. If, in step 602,processing unit 502 determines that the established end-to-endconnection does not satisfy the condition, then the establishedend-to-end connection belongs to the non-first/second group ofestablished end-to-end connections in step 605. Step 604 is omittedbecause there is only one condition.

When conditions are related to performance metrics, determination instep 602 is performed by sending testing data to communications router108 using the established end-to-end connection. The contents of testingdata can be based on, at least in part, contents of data packets,randomly generated contents, or one or more benchmarks testing. The timeperiod to send testing data is preferred to be less than ten seconds asthe number of testing data sent in ten seconds should be enough todetermine the performance, such as round-trip-time, latency and packetdrops in most networks.

Alternatively, instead of using testing data, determination in step 602is done by sending data packets, error correction/detection packets,management packets or health-check packets. Health-check packets aresent for maintenance of the established end-to-end connection and inorder to check the status of the established end-to-end connection. Whendata packets are transmitted via each established end-to-end connectionfor the determination in step 602, after the classification iscompleted, the data packets are only transmitted via specific group(s)of established end-to-end connections. In one of the embodiments, aperformance report is received from communications router 108, and theperformance report is used to determine whether the establishedend-to-end connection satisfies conditions related to performance.Testing data can be transmitted through a plurality of establishedend-to-end connections sequentially, one-by-one, in a group,simultaneously, or almost simultaneously. In one variant, determinationin step 602 is based, at least in part, on information or analysis ofinformation stored in secondary storage 505. Alternatively,determination in step 602 is done by using historical data or historicalperformance data. The historical data is stored in secondary storage505. An example of a scenario where historical data can be used todetermine whether an established end-to-end connection satisfies acondition in step 602 is when the condition is based on a usage metric.For example, an established end-to-end connection satisfies a conditionif its usage has not reached a certain limit, and does not satisfy thecondition if its usage has reached or is close to the certain limit.Therefore the usage needs to be monitored and usage data is stored ashistorical data in secondary storage 505.

In one variant, testing data is sent using OSI layer two packets withPoint-to-point Protocol (PPP), frame relay protocol, Address ResolutionProtocol (ARP), or any other data link layer protocol. Alternatively,testing data is sent using Internet Protocol (IP) packets withtransmission control protocol (TCP), user datagram protocol (UDP), orInternet Control Message Protocol (ICMP). If the established end-to-endconnection uses LTE protocol, testing data is sent using Type 1 or Type2 LTE frames. Alternatively, testing data is sent using Ethernet frames.Alternatively, the format of the testing data is based on, in part, theconditions that are retrieved and the communication protocol of theestablished end-to-end connection.

In one of the embodiments, a score is calculated for the establishedend-to-end connection based on the conditions it satisfies. Theestablished end-to-end connection must satisfy a minimum number ofconditions in order to belong to a group of established end-to-endconnection. The minimum number can be predefined by a user, manufactureror administrator of communications router 106. For example, there arefive conditions and the minimum number of conditions that need to besatisfied by the established end-to-end connection to belong to thefirst group of established end-to-end connections is three. Ifprocessing unit 502 determines that the established end-to-endconnection satisfies at least three conditions, the establishedend-to-end connection belongs to the first group of establishedend-to-end connection. If processing unit 502 determines that theestablished end-to-end connection does not satisfy at least threeconditions, the established end-to-end connection belongs to a non-firstgroup of established end-to-end connections. In one variant, the minimumnumber of conditions that need to be satisfied by the establishedend-to-end connection to belong to a second group of establishedend-to-end connections is two. If processing unit 502 determines thatthe established end-to-end connection does not satisfy at least threeconditions, but the established end-to-end connection satisfies at leasttwo conditions, the established end-to-end connection belongs to thesecond group of established end-to-end connections. There are myriad oftechniques of applying combinations of conditions that require to besatisfied for an established end-to-end connection to belong to acertain group. The combinations can be configured by the user,manufacturer or network administrator of communications router 106according to their preferences. It is beneficial for the user to be ableto configure different combinations of conditions, as establishedend-to-end connections of different characteristics may be required indifferent situations.

Alternatively, weights are assigned to the conditions in order tocalculate a score of the established end-to-end connection. Theclassification of the established end-to-end connection into a group isdone according to the score. For example, there are three conditions,namely, a first condition, a second condition and a third condition. Theweights assigned to the first, second and third conditions are fifty,thirty and twenty respectively. If the established end-to-end connectionsatisfies the first condition only, its score is fifty. If theestablished end-to-end connection satisfies the second condition only,its score is thirty. If the established end-to-end connection satisfiesthe third condition only, its score is twenty. If the establishedend-to-end connection satisfies the first condition and secondcondition, its score is eighty. If the established end-to-end connectionsatisfies the first condition and the third condition, its score isseventy. If the established end-to-end connection satisfies the secondcondition and the third condition, its score is fifty. If theestablished end-to-end connection satisfies all three conditions, itsscore is hundred. A minimum score required to belong to the first groupof established end-to-end connection is fifty. Therefore, if processingunit 502 determines that the score of the established end-to-endconnection is at least fifty according to the weights, the establishedend-to-end connection belongs to the first group of establishedend-to-end connections. The weights are used in order to emphasize oncertain conditions.

In one variant, multiple minimum scores are configured for classifyingthe established end-to-end connections into multiple groups, wherein thescores of the established end-to-end connections are calculatedaccording to weights of conditions they satisfy. Each group is assignedwith a corresponding minimum score, wherein an established end-to-endconnection must have a score more than or equal to the correspondingminimum score assigned to a group in order to belong to the group. Forexample, a first group, a second group, a third group, and a fourthgroup are assigned to corresponding minimum scores of seventy, sixty,fifty, and forty respectively. If an established end-to-end connectionhas a score of seventy or above, it belongs to the first group.Similarly, if an established end-to-end connection has a score betweensixty to seventy, fifty to sixty, or forty to fifty, it belongs to thesecond group, third group, or fourth group respectively.

When some specific conditions are more important to be satisfied thanother conditions, a user can assign higher weights to the specificconditions. Conditions that are more important are given higher priorityby assigning higher weights to the conditions, while the otherconditions that are less important are not completely disregarded.

In one variant, when there is a plurality of established end-to-endconnections, and classification is being conducted for one of theestablished end-to-end connection, communications router 106 does notuse other established end-to-end connections to transmit data. This canreduce interference of other established end-to-end connections andhence the classification is conducted more accurately and effectively.Alternatively, when classification is being conducted for one of theestablished end-to-end connection, communications router 106 continuesusing other established end-to-end connections to transmit data packets.Therefore transmission of data packets is not stopped untilclassification is completed, and hence communications router 106 isalways active in transmitting data packets. This is beneficial becausethis ensures that there's no halt in transmission of data packets everytime classification is conducted.

In one variant, after an established end-to-end connection isclassified, it is reclassified periodically or upon a particular eventoccurs. An established end-to-end connection from one group can beclassified to another group if the established end-to-end connectionshas become satisfying more or fewer conditions. This allowscommunications router 106 to select suitable established end-to-endconnections when network environment changes. This is particularly moreimportant for established end-to-end connections that are carried bywireless technologies as change of physical environment may also changenetwork performance. Particularly, the time period to determine whetheran established end-to-end connection has to be moved to another group orcan stay in the group is in the range of two seconds to two minutes. Thetime period for established end-to-end connections using wiredconnection, is preferred to be longer than the time period forestablished end-to-end connections using wireless connection becauseestablished end-to-end connections using wireless connection usuallyexperience more variations. The more frequent the reclassification isperformed, the earlier communications router 106 is able to move anestablished end-to-end connection from one group to another group upondetecting change of satisfying condition(s). The less frequent thereclassification is performed, the less amount of testing packets needto be transmitted.

For illustration, an established end-to-end connection in the secondgroup can belong to the first group when it satisfies conditions forbelonging to the first group. Similarly, an established end-to-endconnection in the first group is removed from the first group when itfails or stops satisfying conditions for belonging to the first group.Then if the established end-to-end connection satisfies conditions forbelonging to the second group, the established end-to-end connectiongoes from the first group to the second group of established end-to-endconnections. If the established end-to-end connection does not satisfyconditions for belonging to the second group, the established end-to-endconnection goes from the first group to the non-first/second group ofestablished end-to-end connections. Similarly, an established end-to-endconnection in the non-first/second group becomes belonging to the firstgroup when it satisfies conditions to belong to the first group.Similarly, an established end-to-end connection in the non-first/secondgroup becomes belonging to the second group when it satisfies conditionsfor belonging to the second group but does not satisfy conditions forbelonging to the first group. Similarly, an established end-to-endconnection in the second group becomes belonging to the non-first/secondgroup if it fails or stops satisfying conditions required for belongingto the second group. Therefore, the established end-to-end connectionsbelonging to the second group or non-first/second group are performingthe backup or failover functions.

FIG. 6B illustrates a process of classifying established end-to-endconnections into multiple groups based on multiple conditions accordingto one of the embodiments, wherein the multiple conditions are all basedon the same metric. For example, the multiple conditions are based onpacket latency, which is a performance metric. The multiple conditionsare retrieved by processing unit 502 from secondary storage 505. Forbetter illustration, the multiple conditions include a first condition,a second condition, a third condition, a fourth condition, and a fifthcondition. The first, second, third, fourth, and fifth conditions can besatisfied if packet latency is less than 10 ms, 20 ms, 30 ms, 40 ms, and50 ms respectively. The multiple groups include a first group, a secondgroup, a third group, a fourth group and an unsatisfactory group. Instep 611, processing unit 502 determines whether an establishedend-to-end connection satisfies the fifth condition, more particularly,whether the packet latency is less than 50 ms. If the establishedend-to-end connection does not satisfy the fifth condition, moreparticularly, if the packet latency is not less than 50 ms, theestablished end-to-end connection belongs to the unsatisfactory group instep 621. If the established end-to-end connection satisfies the fifthcondition, more particularly, if the packet latency is less than 50 ms,in step 612, processing unit 502 determines whether the establishedend-to-end connection satisfies the fourth condition, more particularly,whether the packet latency is less than 40 ms. If the establishedend-to-end connection does not satisfy the fourth condition, moreparticularly, if the packet latency is not less than 40 ms, theestablished end-to-end connection belongs to the fifth group in step622. If the established end-to-end connection satisfies the fourthcondition, more particularly, if the packet latency is less than 40 ms,in step 613, processing unit 502 determines whether the establishedend-to-end connection satisfies the third condition, more particularly,whether the packet latency is less than 30 ms. If the establishedend-to-end connection does not satisfy the third condition, moreparticularly, if the packet latency is not less than 30 ms, theestablished end-to-end connection belongs to the fourth group in step623. If the established end-to-end connection satisfies the thirdcondition, more particularly, if the packet latency is less than 30 ms,in step 614, processing unit 502 determines whether the establishedend-to-end connection satisfies the second condition, more particularly,whether the packet latency is less than 20 ms. If the establishedend-to-end connection does not satisfy the second condition, moreparticularly, if the packet latency is not less than 20 ms, theestablished end-to-end connection belongs to the third group in step624. If the established end-to-end connection satisfies the secondcondition, more particularly, if the packet latency is less than 20 ms,in step 615, processing unit 502 determines whether the establishedend-to-end connection satisfies the first condition, more particularly,whether the packet latency is less than 10 ms. If the establishedend-to-end connection does not satisfy the first condition, moreparticularly, if the packet latency is not less than 10 ms, theestablished end-to-end connection belongs to the second group in step625. If the established end-to-end connection satisfies the firstcondition, more particularly, if the packet latency is less than 10 ms,the established end-to-end connection belongs to the first group in step626.

Alternatively, the process of FIG. 6B can be implemented using multipleconditions that are based on something other than packet latency, suchas other performance metrics, usage price, IP address range, or anyother metric that can be used to implement multiple thresholds.

The steps of FIG. 6B may be conducted sequentially, simultaneously, oralmost simultaneously. The order of the steps can be different than theorder illustrated in FIG. 6B.

FIG. 6C illustrates a process of classifying established end-to-endconnections into multiple groups based on multiple conditions accordingto one of the embodiments. The multiple conditions are stored insecondary storage 505. For illustration purpose, the multiple conditionsinclude a first condition, a second condition, a third condition, afourth condition and a fifth condition. A plurality of establishedend-to-end connections is classified into multiple groups in step 630.If processing unit 502 determines that an established end-to-endconnection satisfies the first condition, the established end-to-endconnection belongs to first group 631. If processing unit 502 determinesthat an established end-to-end connection satisfies the secondcondition, the established end-to-end connection belongs to second group632. If processing unit 502 determines that an established end-to-endconnection satisfies the third condition, the established end-to-endconnection belongs to third group 633. If processing unit 502 determinesthat an established end-to-end connection satisfies the fourthcondition, the established end-to-end connection belongs to fourth group634. If processing unit 502 determines that an established end-to-endconnection satisfies the fifth condition, the established end-to-endconnection belongs to fifth group 635.

In one variant, the multiple conditions are based on multiple metrics,criteria, and/or parameters. For example, the first condition is basedon location, the second condition is based on time, the third conditionis based on a user identity, the fourth condition is based on security,and the fifth condition is based on communication protocol. Anestablished end-to-end connection may belong to more than one group ifit satisfies more than one condition. Alternatively, the multipleconditions are based on the same metric, criterion or parameter, butmultiple threshold values. For example, the multiple conditions arebased on throughput, which is a performance metric. The first conditionis satisfied if throughput value of the established end-to-endconnection is within a first range. Similarly, the second, third, fourthor fifth conditions are satisfied if the throughput value of theestablished end-to-end connection is within a second, third, fourth andfifth range respectively. More particularly, the first, second, third,fourth and fifth ranges are all different. In another alternative, theprocess of FIG. 6C can be implemented using multiple conditions that arebased on something other than throughput, such as other performancemetrics, usage price, IP address range, or any other metric, criterionor parameter that can be used to implement multiple numeric thresholds.

In one of the embodiments, established end-to-end connections areclassified into multiple groups based on combinations of conditions thatare satisfied by the established end-to-end connections. For example,established end-to-end connections satisfying a first and secondcondition belong to one group. Established end-to-end connectionssatisfying the first, second and third conditions belong to anothergroup. Similarly, end-to-end connections satisfying other combinationsof conditions belong to other groups. Having multiple groups ofestablished end-to-end connections can be beneficial when packets ofspecific data types need to be transmitted using specific establishedend-to-end connections satisfying specific combinations of conditions,wherein the specific data types of the packets can be detected usingmyriad techniques for packet inspection, including deep packetinspection.

In one of the embodiments, an established end-to-end connection isclassified into a certain group based on conditions not satisfied by theestablished end-to-end connection. For example, a condition is based onuser identity which specifies a first user. When a user of anestablished end-to-end connection is determined to not be the firstuser, the established end-to-end connection belongs to a first group.When a user of an established end-to-end connection is determined to bethe first user, the established end-to-end connection does not belong tothe first group.

In one of the embodiments, after a plurality of established end-to-endconnections are classified into groups by communications router 106,established end-to-end connections not being used to transmit datapackets are maintained and not disconnected. In some communicationprotocols, if an established end-to-end connection is not used totransmit packets for a certain period of time, the establishedend-to-end connection is disconnected. In order to avoid this,maintenance packets, health check packets or management packets aretransmitted through the established end-to-end connections in order tomaintain the established end-to-end connection and keep the establishedend-to-end connection established. This ensures that the establishedend-to-end connection is available for use whenever it needs to be usedto transmit data packets by the communications router withoutexperiencing slowness.

Description of Conditions:

In one of the embodiments, classification of established end-to-endconnections into different groups is based on whether the establishedend-to-end connections satisfy certain conditions. The conditions areselected from a group consisting of performance metric, serviceprovider, location, time, usage price, security, user identity, InternetProtocol (IP) address range, communication protocol, communicationtechnology, application, and device. The performance metric can be basedon one or more of throughput, error rates, packet latency, packetjitter, symbol jitter, quality of service, bandwidth, bit error rate,packet error rate, frame error rate, dropped packet rate, queuing delay,round trip time, capacity, signal level, interference level, bandwidthdelay product, handoff delay time, signal-to-interface ratio, andsignal-to-noise ratio.

Service Provider:

In one of the embodiments, a condition is based on, at least in part,the service provider of the established end-to-end connection. Forexample, service provider A provides more reliable service than serviceprovider B. Then, a condition can be such that an established end-to-endconnection can satisfy the condition if it is from service provider A.The user can have a preference of service provider A over other serviceproviders due to reliability, costs, or performance.

Usage Metric:

In one of the embodiments, a condition is based on, at least in part,the usage metric which specifies a usage limit for the establishedend-to-end connection. For example, an established end-to-end connectionsatisfies the condition if its usage has not reached the usage limit,and does not satisfy the condition if its usage has reached or is closeto the usage limit. Most service providers charge a low price perGigabyte as long as the usage is below a certain threshold, and when theusage goes above the certain threshold, the price per Gigabyteincreases. Therefore, by making the usage limit equal to the certainthreshold, cost of using the established end-to-end connection is keptwithin a budget.

Location:

In one of the embodiments, a condition is based on, at least in part,the location. For example, when communications router 106 is in acertain location, using some established end-to-end connections may bemore preferable than other established end-to-end connections. The morepreferable end-to-end connections satisfy the condition. It isbeneficial to have a condition based on the location because someestablished end-to-end connections can be more preferable in certainlocations mainly due to higher signal strengths of the establishedend-to-end connections in the certain locations. In another example,when communications router 106 is in a certain location using a firstestablished end-to-end connection using LTE protocol, and a secondestablished end-to-end connection using Wi-Fi connection becomesavailable in the certain location, communications router 106 can switchfrom the first established end-to-end connection to the secondestablished end-to-end connection and perform network offloading basedon the location. Therefore, in the certain location, the firstestablished end-to-end connection does not satisfy the condition and thesecond established end-to-end connection satisfies the condition.

Time:

In one of the embodiments, a condition is based on, at least in part,the time. For example, at a certain time in the day, some establishedend-to-end connections may be more preferable than other establishedend-to-end connections. The more preferable end-to-end connectionssatisfy the conditions. This embodiment makes use of differences incost, performance and reliability of established end-to-end connectionsduring peak and off-peak hours.

Usage Price:

In one of the embodiments, a condition is based on, at least in part,the usage price. If the price of using an established end-to-endconnection is more than a user's price limit, the established end-to-endconnection does not satisfy the condition. If the price of using theestablished end-to-end connection is equal to or less than the user'sprice limit, the established end-to-end connection satisfies thecondition. Setting a price limit is beneficial for users having a costbudget for using established end-to-end connections. For example, theuser's price limit is $10 per Gigabyte. Therefore, if the price of usingan established end-to-end connection is more than $10 per Gigabyte, theestablished end-to-end connection does not satisfy the condition, and ifthe price of using the established end-to-end connection is less than orequal to $10 per Gigabyte, the established end-to-end connectionsatisfies the condition.

Security:

In one of the embodiments, a condition is based on, at least in part,the security. If an established end-to-end connection is not secureenough, or the security protocol used in the established end-to-endconnection is not a security protocol preferred by the user, theestablished end-to-end connection does not satisfy the condition. Havinga secure established end-to-end connection is extremely important inorder to achieve authenticity, integrity and confidentiality of datapackets transmitted through the established end-to-end connection.Integrity of the data packets can maintain and assure the accuracy andconsistency of information provided in the data packets. Confidentialityof the data packets is also required in order to prevent disclosure ofinformation provided in the data packets to unauthorized individuals orsystems. For example, the condition is that an established end-to-endconnection has to use the encryption standard AES 192. If an establishedend-to-end connection uses AES 128 encryption, or any encryptionstandard other than AES 192, the established end-to-end connection doesnot satisfy the condition. If the established end-to-end connection usesAES 192 encryption, the established end-to-end connection satisfies thecondition. Alternatively, the established end-to-end connection alsosatisfies the condition if the encryption standard used by theestablished end-to-end connection provides higher security measures thanAES 192, such as AES 256.

User Identity:

In one of the embodiments, a condition is based on, at least in part,the user identity. For example, user authentication is required toaccess the established end-to-end connection. According to the identityof users, certain established end-to-end connections may be reserved forcertain users. If the established end-to-end connection is reserved forthe user using communications router 106, then the establishedend-to-end connection satisfies the condition. It is beneficial to havea condition based on the user identity in a communications router thathas more than one user, wherein each user may have different preferencesfor using established end-to-end connections.

IP Address Range:

In one of the embodiments, a condition is based on, at least in part,the IP address range. For example, a range of IP addresses is specifiedin the condition, such that, if an established end-to-end connection isbetween communications router 106 and a network interface whose IPaddress does not belong to the range of IP addresses, the establishedend-to-end connection does not satisfy the condition. If the establishedend-to-end connection is between communications router 106 and a networkinterface whose IP address belongs to the range of IP addresses, theestablished end-to-end connection satisfies the condition.

Communication Protocol:

In one of the embodiments, a condition is based on, at least in part,the communication protocol. Communication protocols include PPP, framerelay protocol, ARP and IP such as TCP, UDP or ICMP. For example, a userwants to use established end-to-end connections to transmit a certaintype of data packet with a specific communication protocol. If anestablished end-to-end connection uses the specific communicationprotocol, then the established end-to-end connection satisfies thecondition. If the established end-to-end connection does not use thespecific communication protocol, then the established end-to-endconnection does not satisfy the condition. For example, the user wantsto use an established end-to-end connection for transmitting PPP datapackets. If the established end-to-end connection uses PPP, theestablished end-to-end connection satisfies the condition. If theestablished end-to-end connection does not use PPP, the establishedend-to-end connection does not satisfy the condition.

Communication Technology:

In one of the embodiments, a condition is based on, at least in part,the communication technology. Communication technologies includewireless technologies, Wi-Fi, WiMax, High-Speed Packet Accesstechnology, 3GPP Long Term Evolution (LTE) or the like. For example, auser wants to use established end-to-end connections to transmit acertain type of data packet with a specific communication technology. Ifan established end-to-end connection uses the specific communicationtechnology, then the established end-to-end connection satisfies thecondition. If the established end-to-end connection does not use thespecific communication technology, then the established end-to-endconnection does not satisfy the condition. For illustration purposes,the user wants to use an established end-to-end connection to transmitdata packets with 3GPP LTE technology. Therefore, if an establishedend-to-end connection uses 3GPP LTE, the established end-to-endconnection satisfies the condition. It is beneficial to have a conditionbased on the communication technology because of the varying features ofdifferent communication technologies, wherein some specific features ofcertain communication technologies can be required to fulfill thepurpose of using an established end-to-end connection.

Application:

In one of the embodiments, a condition is based on, at least in part,the application. For example, the user wants to use establishedend-to-end connections to transmit data packets to a certainapplication. If an established end-to-end connection is suitable forbeing used in the certain application, the established end-to-endconnection satisfies the condition. Alternatively, when an establishedend-to-end connection is known to be suitable for a certain application,the established end-to-end connection satisfies the condition only ifthe user wants to use the established end-to-end connection fortransmitting data packets to the certain application.

Device:

In one of the embodiments, a condition is based on, at least in part,the device. For example, the user wants to use established end-to-endconnections formed between network interfaces that use a specific typeof modem of a specific model number. If an established end-to-endconnection is formed between network interfaces using the specific typeof modem of the specific model number, then the established end-to-endconnection satisfies the condition. This can be beneficial in caseswhere the performance or usage price or reliability of the specific typeof modem of the specific model number is better and more suitable fortransmitting the type of data packet that the user wants to transmitthrough the established end-to-end connection. For example, the userwants to use established end-to-end connections that are establishedwith a Verizon Wireless 4G LTE USB Modem 551L. If the establishedend-to-end connection connects communications router 106 to a VerizonWireless 4G LTE USB Modem 551L, then the established end-to-endconnection satisfies the condition. In another example, the user wantsto use established end-to-end connections using a Subscriber IdentityModule (SIM) card subscribed to a specific service provider or with aspecific service plan. If an established end-to-end connection uses aSIM card from the specific service provider or with the specific serviceplan, the established end-to-end connection satisfies the condition.This can be beneficial in terms of costs and reliability associated withthe SIM card.

Ranking of the Groups:

In one of the embodiments of the present invention, when establishedend-to-end connections are classified into different groups, such asfirst group, second group and non-first/second group, ranks are assignedto the different groups in order to prioritize transmitting packetsusing established end-to-end connections in groups with higher ranks.This prioritization is important so that the established end-to-endconnections with the higher ranks and the best characteristics accordingto the user's preferences are used to send data packets. Other types ofpackets, such as management packets, error correction packets, and allnon-data packets are transmitted using established end-to-endconnections with lower ranks. Therefore, established end-to-endconnections with higher ranks are reserved for data packets, and theircapacity is not filled by other types of packets. Alternatively,established end-to-end connections with higher ranks are also be used totransmit other types of packets as some administrators may considermanagement packets and error correction packets should have the same oreven higher priority than data packets. Established end-to-endconnections belonging to a particular group have the same rank. In onevariant, rank of the first group of established end-to-end connectionsis the highest. Rank of non-first group of established end-to-endconnections is lower than rank of the first group of establishedend-to-end connections. If there is a second group of establishedend-to-end connections, rank of the second group of establishedend-to-end connection is the second-highest. Rank of thenon-first/second group of established end-to-end connection is thelowest.

Alternatively, there are multiple groups of establishedend-to-end-connections, wherein the multiple groups of establishedend-to-end connections are created based on conditions satisfied or notsatisfied by established end-to-end connections. FIG. 6B and FIG. 6Cillustrate in detail how established end-to-end connections can beclassified into multiple groups.

In one variant, communications router 106 transmits data packets tocommunications router 108 using only the first group of establishedend-to-end connections. When the capacity of the first group ofestablished end-to-end connections is full, the data packets aretransmitted later. Communications router 106 waits till the first groupof established end-to-end connections has enough capacity, and thentransmits the data packets to communications router 108 using the firstgroup of established end-to-end connections. There can be variousreasons for doing this. For example, the first group of establishedend-to-end connections is more reliable and faster than the second groupor non-first/second group of established end-to-end connections.Therefore, the user may choose to always use the first group ofestablished end-to-end connections to transmit data packets due to theirreliability and speed.

Alternatively, communications router 106 transmits data packets tocommunications router 108 using only the first group of establishedend-to-end connections. When the capacity of the first group ofestablished end-to-end connections is full, communications router 106uses the non-first group of established end-to-end connections totransmit data packets to communications router 108.

Alternatively, communications router 106 transmits data packets tocommunications router 108 using only the first group of establishedend-to-end connections. When the capacity of the first group ofestablished end-to-end connections is full, communications router 106uses the second group of established end-to-end connections to transmitdata packets to communications router 108. When the capacity of thesecond group of established end-to-end connections is also full, thedata packets are not transmitted until capacity is available at thefirst group or the second group of established end-to-end connections.Communications router 106 waits till the first group or second group ofestablished end-to-end connections has enough capacity, and thentransmits the data packets to communications router 108 using the firstgroup of established end-to-end connections. It should be noted that thesecond group of established end-to-end connections is only used whencapacity of the first group of established end-to-end connections stillfull.

Alternatively, when the capacity of the first group of establishedend-to-end connections is full, communications router 106 uses thesecond group of established end-to-end connections to transmit datapackets to communications router 108. When the capacity of the secondgroup of established end-to-end connections is also full, communicationsrouter 106 uses the non-first/second group of established end-to-endconnections to transmit the data packets to communications router 108.The non-first/second group of established end-to-end connectionscomprises the non-first group and non-second group of establishedend-to-end connections.

In one of the embodiments, established end-to-end connections in thesecond group are ranked according to the conditions satisfied by them.The first group of established end-to-end connections satisfies allconditions and the non-first/second group of established end-to-endconnections do not satisfy any conditions. However, the second group ofestablished end-to-end connections satisfies some conditions, but notall conditions. Therefore, the second group may comprise establishedend-to-end connections of many characteristics, wherein each establishedend-to-end connections may or may not satisfy the same conditions. Sincethe established end-to-end connections in the second group havedifferent characteristics, it is beneficial to assign ranks to eachestablished end-to-end connection so that packets can be transmittedusing established end-to-end connections with higher ranks according tothe user's preferences.

FIG. 7 illustrates rankings of multiple groups of established end-to-endconnections at a communications router according to one of theembodiments. The rankings are stored in secondary storage 505 and areimplemented by processing unit 502. Ranking table 700 lists multiplegroups and their respective ranks. In one variant, ranking 720 is takeninto consideration. According to ranking 720, first group 711, secondgroup 712, third group 713, fourth group 714 and fifth group 715 areassigned with first rank, second rank, third rank, fourth rank and fifthrank respectively. Rankings can be configured by the user, or networkadministrator of the communications router. Hence in another variant,when ranking 730 is taken into consideration, first group 711, secondgroup 712, third group 713, fourth group 714 and fifth group 715 areassigned with second rank, third rank, first rank, fourth rank, andfifth rank respectively. It would be appreciated by those skilled in thearts that assigning ranks to each group of established end-to-endconnections is beneficial in order to prioritize transmitting packetsusing established end-to-end connections in groups with higher ranks. Itwould also be appreciated that different users may have differentpreferences when assigning ranks to groups, and hence for illustrationpurposes, two kinds of rankings, namely ranking 720 and ranking 730 areshown in ranking table 700.

Transmitting

According to one of the embodiments of the present invention, when anestablished end-to-end connection fails a condition specific to a datatype, communications router 106 does not transmit data packets belongingto the data type through the established end-to-end connection but stilltransmits data packets not belonging to the data type through theestablished end-to-end connection. The established end-to-end connectionthat is used to transmit a certain type of data packets but not alltypes of data packets belongs to a second group of establishedend-to-end connection.

The use of second group of established end-to-end connections totransmit data packets may help improve data packet transmissionperformance as not all established end-to-end may have the same networkperformance. For example, some established end-to-end connections mayhave larger latency and some established end-to-end connections may havefewer packet drops. Depending on which conditions an establishedend-to-end connection meets, the established end-to-end connection maybelong to a group of established end-to-end connections that aresuitable for one kind of data packets but not suitable for another kindof data packets. For example the second group of established end-to-endconnections with high-latency is suitable for data packets belonging tofile transfer application, but not suitable for data packets belongs tovoice communications.

According to one of the embodiments of the present invention,communications router 106 keeps monitoring if any of the second group ofestablished end-to-end connections has become eligible to belong to thefirst group of established end-to-end connections by satisfying allconditions or has become eligible to belong to the non-first/secondgroup of established end-to-end connections by failing to satisfy otherconditions. Moving an established end-to-end connection from the secondgroup of established end-to-end connection to the first group ofestablished end-to-end connection may improve the overall networkperformance of the aggregated end-to-end connection becausecommunications router 106 has more choices of established end-to-endconnections in the first group to transmit data packets. Also, moving anestablished end-to-end connection from the second group of establishedend-to-end connection to the non-first/second group of establishedend-to-end connection may improve the overall network performance of theaggregated end-to-end connection because communications router 106 doesnot transmit data packets through established end-to-end connections inthe non-first/second group to reduce data packet loss, data packetdelay, data packet jitter and other undesired effect.

Although it is important that communications router 106 does nottransmit data packets from communications router 106 to communicationsrouter 108 through the non-first/second group of established end-to-endconnections for certain applications, the stringent requirement of notusing the non-first/second group of established end-to-end connectionscan be relaxed by allowing communications router 106 to use the secondgroup of established end-to-end connections for other applications inorder to improve the data packet performance.

For applications using TCP for communication, data packets drop in oneof the established end-to-end connections, of the aggregated end-to-endconnection, belonging to the first group may negatively affect theoverall throughput of the aggregated end-to-end connection. This isbecause data packets of one TCP session transmitted by communicationsrouter 106 to communications router 108 are being spread among the firstgroup of established end-to-end connections. As a result, when anestablished end-to-end connection belonging to the first group becomesan established end-to-end connection belonging to the non-first/secondgroup, communications router 106 should not transmit data packetsthrough the established end-to-end connections belonging to thenon-first/second group.

Packet Inspection

According to one of the embodiments, packets are distributed accordingto three categories, namely, data packets, management packets, and errorcorrection packets. In one alternative, distribution of the packets isdone by packet inspection where the content of the packets are inspectedto determine whether they are data packets, management packets, or errorcorrection packets. In another alternative, for packets that originatefrom communications router 106, for example management packets, packetinspection is not required to determine which category the packetsbelong to.

Data packets are packets that carry information that needs to betransmitted from a communications router 106 to a communications router108 via at least one established end-to-end connection. Data packets maybe originated from communications router 106 or may be originated fromhost(s) or node(s) connected to communications router 106. Destinationof the data packets is communications router 108 host(s) and/or node(s)connected to communications router 108.

Management packets are packets that are created by communications router106 and transmitted via established end-to-end connections in order tomanage the aggregated end-to-end connection. Management packets includerouting packets, switching packets, health-check packets, maintenancepackets, performance report packets, and other types of packets requiredto manage aggregated end-to-end connections. Management information isused to assist communications router 106, including to manage theestablished end-to-end connections, to manage the aggregated end-to-endconnections, to test the performance of an end-to-end connection and tocommunicate with communications router 108 to coordinately manage theestablished end-to-end connections. The management information can bepiggybacked with data in one or more data packets or can be transmittedas one or more standalone data packets.

Error correction packets are packets that are transmitted bycommunications router 106 to correct or detect any errors in datapackets that are transmitted from communications router 106 tocommunications router 108. Error correction packets include Forwarderror correction (FEC) packets or Automatic repeat request (ARQ)packets, Parity packets, or the like.

According to a preferred embodiment, the first group of establishedend-to-end connections is used to transmit data packets. The first groupof established end-to-end connections has the highest rank amongst otherestablished end-to-end connections. Therefore the first group ofestablished end-to-end connections is most suitable for transmittingdata packets according to the user's preferences. Management packets,error correction packets are transmitted using the second group ofestablished end-to-end connections or the non-first/second group ofestablished end-to-end connections. The benefit of transmittingmanagement packets and error correction packets using the non-firstgroup and/or the non-first/second group of established end-to-endconnections is that the first group of established end-to-endconnections is reserved for data packets and higher capacity isavailable for transmitting data packets. Furthermore, it is preferableto transmit error correction packets using different establishedend-to-end connections than the ones used for transmitting data packetsso that the same errors are not found in the error correction packets.

In one of the embodiments, if capacity of the first group of establishedend-to-end connections is full, data packets are transmitted using thesecond group of established end-to-end connections. In one of theembodiments, if capacity of the first group and the second group ofestablished end-to-end connections is full, data packets are transmittedusing the non-first/second group of established end-to-end connections.

In one of the embodiments, if the first group of established end-to-endconnections has enough capacity for transmitting non-data packets inaddition to data packets, then management packets are transmitted usingthe first group of established end-to-end connections. In one of theembodiments, management packets for authentication of a user aretransmitted using the first group of established end-to-end connections.For example, established end-to-end connections using File TransferProtocol (FTP) require authentication of users, and therefore, theauthentication information is transmitted using the first group ofestablished end-to-end connections.

In one of the embodiments, data packets are distributed amongestablished end-to-end connections within a group, and transmittedthrough the established end-to-end connections using techniquesdescribed in FIGS. 2A and 2B.

Failover

According to one of the embodiments of the present invention describedin FIG. 8 and the classification illustrated in FIG. 4, when one or moreestablished end-to-end connections belonging to the first group failsand other established end-to-end connections belonging to the firstgroup does not have enough capacity to transmit data packets,communications router 106 uses one or more established end-to-endconnections belonging to the second group to transmit the data packets.Therefore, the established end-to-end connections belonging to thesecond group are performing the backup or failover functions. There aremany possibilities why the established end-to-end connection belongingto the first group does not have enough capacity. For example, thenumber of data packets is too large. In another example, an establishedend-to-end connection belonging to the first group fails bydisconnection or equipment malfunctioning such that the establishedend-to-end connection breaks and cannot be classified to be in the firstgroup anymore. Although the performance of the established end-to-endconnection belonging to the second group may not be as good as those inthe first group, the established end-to-end connection belonging to thesecond group can provide additional capacity for transmitting datapackets.

According to one of the embodiments of the present invention, a firstestablished end-to-end connection is the only established end-to-endconnection belonging to the first group. Data packets are transmittedthrough the first established end-to-end connection in step 801. Datapackets are continued to be transmitted through the first establishedend-to-end connection in step 801 if processing unit 502 determines thatnumber of data packet drops or losses detected in the first establishedend-to-end connection within a first predefined time period is fewerthan a first predefined number of data packet drops or losses in step802. When processing unit 502 detects at least the first predefinednumber of data packet drops or losses in the first establishedend-to-end connection within the first predefined time period in step802, communications router 106 transmits “warm-up” packets through asecond established end-to-end connection, which belongs to the secondgroup, in step 803. Data packets are continued to be transmitted throughthe first established end-to-end connection in step 801 if processingunit 502 determines that number of data packet drops or losses detectedin the first established end-to-end connection within a secondpredefined time period is fewer than a second predefined number of datapacket drops or losses in step 804. When processing unit 502 detects atleast an additional second predefined number of data packet drops orlosses in the first established end-to-end connection within the secondpredefined time period in step 804, communications router 106 startstransmitting data packets through the second established end-to-endconnection in step 805. After a third predefined time period,communications router 106 tests the performance of the first establishedend-to-end connection to check if there is still at least a thirdpredefined number of packet drops or losses within a fourth predefinedtime period in step 806. If the first established end-to-end does notexperience at least the third predefined number of packet drops orlosses within the fourth predefined time period, communications router106 starts reusing the first established end-to-end connection totransmit data packets in step 807 and stops using the second establishedend-to-end connection to transmit data packets in step 808. Ifprocessing unit 502 detects at least the third predefined number of datapacket drops or losses in the first established end-to-end connectionwithin the fourth predefined time period in step 806, data packets arecontinued to be transmitted through the second established end-to endconnection in step 805. It is important to allow the third predefinedtime period to pass before conducting step 806 so that the switchbetween using the first established end-to-end connection and using thesecond established end-to-end connection is not too frequent as this maydeteriorate the overall performance. In one variant, step 808 is omittedand communications router continues transmitting data packets, moreparticularly “warm-up” packets, through the second establishedend-to-end connection so that the second established end-to-endconnection remains activated. The process of FIG. 8 is performedperiodically.

The “warm-up” packets are used in step 803 to assist the secondestablished end-to-end connection to be fully activated in time. Asdifferent access service providers deploy different mechanism to handledata packets from inactive connections, established end-to-endconnections that use inactive connections may experience slowness whendata packets start being transmitted through them. Also, differentnetwork adapter manufacturers may deploy different techniques inmanaging network interfaces that are inactive. For example, wirelessmodem, being used as a network interface or part of a network interface,may be switched into an energy-saving mode or sleep mode after a longperiod of receiving or transmitting no packet. In order to reduce theprobability of slowness experienced by the data packets, communicationsrouter 106 transmits “warm-up” packets to make the connections becomeactive. This is especially more important for wireless connections, suchas connections using satellite, 3G, LTE and cellular technologies, thatare used for establishing end-to-end connections. The “warm-up” packetsmay contain data or may be health-check packets. The size and frequencyof sending the “warm-up” packets should be adjusted for differentcommunication technologies and protocols that are used to establish theend-to-end connection.

Another purpose of transmitting the “warm-up” packets in step 803 is totest whether the second established end-to-end connection is still inoperation and still belong to the second group. If the secondestablished end-to-end connection is not in operation or should notremain in the second group, a third established end-to-end connectionwhich belongs to the second group is selected and is tested with“warm-up” packets in step 803.

The first predefined number of data packet drops or losses, secondpredefined number of data packet drops or losses, and third predefinednumber of packet drops or losses are set by the manufacturer,administrator or user of communications router 106. These predefinednumbers can be the same or different. In one preferred embodiment, thefirst predefined first number of data packet drops or losses is lessthan five, the second predefined first number of data packet drops orlosses is less than ten and the second predefined first number of datapacket drops or losses is less than five. The smaller a predefinednumber is, the performance requirement of the established end-to-endconnection is more stringent because the number of packets allowed to bedropped or lost is smaller.

The first predefined time period, the second predefined time period, thethird predefined time period and the fourth predefined time period areset by the manufacturer, administrator or user of communications router106. These predefined time periods can be the same or different. Thelonger the first predefined time period, the second predefined timeperiod, and the fourth predefined time period are, the performancerequirement of the established end-to-end connection is more stringentbecause the allowed packet dropping rate or packet losing rate issmaller.

According to one of the embodiments of the present invention and theclassification illustrated in FIG. 4, communications router 106transmits data packets through a first established end-to-endconnection. When the probability of the first established end-to-endconnection not being able to remain in the first group reaches apredefined threshold or the first established end-to-end connection isabout to be removed from the first group, communications router 106transmits “warm-up” packets through a second established end-to-endconnection, which belongs to the second group. When the firstestablished end-to-end connection no longer remains in the first group,communications router 106 starts transmitting data packets through thesecond established end-to-end connection. After a predefined timeperiod, when the first established end-to-end connection is in the firstgroup again, communications router 106 use the first establishedend-to-end connection to transmit data packets again and stop using thesecond established end-to-end connection to transmit data packets.

Communications router 106 determines the probability, of whether thefirst established end-to-end connection can remain in the first group,based on how close the conditions of the first group are to be violated.For example, the conditions of the first group are that average latencyis below 50 ms, average packet loss rate is below one packet in twothousand packets, and no data transmission between five o'clock and sixo'clock in the afternoon respectively. For illustration purpose, whenthe first established end-to-end connection experiences average latencyof rises from 40 milliseconds to 48 milliseconds in ten seconds, theprobability may raise to a very high value depending on the methodologyto calculate the probability. In such case, communications router 106transmits “warm-up” packets through the second established end-to-endconnection as the first established end-to-end connection may be removedfrom the first group soon. Also for illustration purpose, when the timeis four o'clock and fifty-eight minutes in the afternoon, communicationsrouter 106 transmits “warm-up” packets through the second establishedend-to-end connection to test whether the second established end-to-endconnection still belongs to the second group. If the second establishedend-to-end connection still belongs to the second group, communicationsrouter 106 start transmitting data packet through the second establishedend-to-end connection when the time is just about at five o'clock in theafternoon because the first established end-to-end connection will failthe conditions of the first group at five o'clock in the afternoon.

According to one of the embodiments of the present invention, when thesecond established end-to-end connection no longer belongs to the secondgroup and also does not belong to the first group, the secondestablished end-to-end connection is either broken or is classified intonon-first/second group. Therefore, communications router 106 transmitsanother set of “warm-up” packets through another one or more establishedend-to-end connections belonging to the second group to determine whichestablished end-to-end connection can be used in case the firstestablished end-to-end connection no longer remains in the first group.In case there are no more established end-to-end connections inbelonging to the second group, communications router 106 transmitsanother set of “warm-up” packets through one or more establishedend-to-end connections belonging to the non-first/second group. As thesecond established end-to-end connection is determined to be broken orto be in non-first/second group, the window of time to determine towhich established end-to-end connection can be used has shrank as ittakes time for using “warm-up” packets to determine the quality and/orstatus of other established end-to-end connections.

The first group of established end-to-end connections can be determinedby using the embodiments described above.

FIG. 9 is a flowchart illustrating a process according to one of theembodiments. An aggregated end-to-end connection is established in step901. The aggregated end-to-end connection comprises a plurality ofestablished end-to-end connections. The plurality of establishedend-to-end connections comprise a first established end-to-endconnection classified into a first group of established end-to-endconnections, and a second established end-to-end connection classifiedinto a second group of established end-to-end connections. In step 902,processing unit 502 determines whether a first condition is satisfied.If the first condition is satisfied, data packets are transmittedthrough the first established end-to-end connection in step 905. If thefirst condition is not satisfied, in step 904, the first establishedend-to-end connection is not used to transmit data packets. In step 903,processing unit 502 determines whether a second condition is satisfied.If the second condition is satisfied, data packets are transmittedthrough the second established end-to-end connection in step 907. If thesecond condition is not satisfied, the second established end-to-endconnection is not used to transmit data packets in step 906.

According to one of the embodiments of the present invention, theplurality of established end-to-end connections are classified intogroups after step 901 and before determining whether the first andsecond conditions are satisfied. The classification is used to determinewhich established end-to-end connections should be grouped together.Therefore a condition can be applied to all established end-to-endconnections belonging to the same group. Further, the same group ofestablished end-to-end connections can be managed together.

In one variant, the first condition and the second condition aremutually exclusive. For example, the first condition is satisfied whenthe second condition is not satisfied, and the second condition issatisfied when the first condition is not satisfied. Therefore, datapackets are transmitted either through the first established end-to-endconnection in step 906 or through the second established end-to-endconnection in step 907.

Alternatively, the first condition and the second condition are notrelated. It is possible that both first and second conditions aresatisfied at the same time. When both the first condition and the secondcondition are satisfied, data packets are distributed among andtransmitted through both the first and second established end-to-endconnections.

Alternatively, the first condition and the second condition are related.Such that probability of satisfying a condition is affected by whetheranother condition is satisfied or not satisfied.

In one variant, when the first condition is satisfied in step 902, datapackets are transmitted through all established end-to-end connectionsbelonging to the first group in step 905 and when the first condition isnot satisfied in step 902, data packets are not transmitted through anyestablished end-to-end connections belonging to the first group in step906. Similarly, when the second condition is satisfied in step 903, datapackets are transmitted through all established end-to-end connectionsbelonging to the second group in step 907, and when the second conditionis not satisfied in step 903, data packets are not transmitted throughany established end-to-end connections belonging to the second group instep 906.

In one variant, the first condition and second condition can be based onperformance of certain established end-to-end connections while usingthe certain established end-to-end connections, service provider ofcertain established end-to-end connections, location of communicationsrouter 106, time of the day, usage price of certain establishedend-to-end connections, security protocols of certain establishedend-to-end connections, user identity, IP address range of destinationof the aggregated end-to-end connection, communication protocol ofcertain established end-to-end connections, communication technology ofcertain established end-to-end connections, application using certainestablished end-to-end connections, and device using certain establishedend-to-end connections.

Failure of Network Interface

Network Interface with End-to-End Connection

According to one of the embodiments of the present invention and theclassification illustrated in FIG. 4, communications router 106 has aplurality of network interfaces and some of the network interfaces areconnected with connections. Communications router 106 establishes afirst established end-to-end connection using a first network interfacethrough a first connection. The first established end-to-end connectionis classified into the first group by processing unit 502.Communications router 106 does not have other established end-to-endconnections but have possible end-to-end connections that can be usedthrough at least one non-first network interface. When processing unit502 determines that the probability of the first established end-to-endconnection not being able to remain in the first group has reached apredefined threshold, the first established end-to-end connection isabout to be removed from the first group, or the network interface hasreported at least one error that may cause the first establishedend-to-end connection to not remain in the first group, communicationsrouter 106 activates a second network interface and tries to establish asecond end-to-end connection with communications router 108. If thesecond end-to-end connection can be established successfully and satisfythe condition(s) of the second group, processing unit 502 classifies thesecond end-to-end connection into the second group. If the secondend-to-end connection cannot be established successfully and/or cannotsatisfy the condition(s) of the second group, communications router 106activates other non-first/second network interface(s) and processingunit 502 determines which network interface can establish an end-to-endconnection that can satisfy the condition(s) of the second group.

Once there is an established end-to-end connection that can satisfycondition(s) of the second group, communications router 106 is ready totransmit data packet using the established end-to-end connection if thefirst established end-to-end connection no longer belongs to the firstgroup. This embodiment does not require a second established end-to-endconnection to be established in advance. Instead the second establishedend-to-end connection is established when the first end-to-endconnection has a high probability or about not to remain in the firstgroup. This embodiment reduces the needs to send packets throughnon-first network interface(s) and result in fewer uses of connectionsthat are connected to non-first network interface(s).

For those who are skilled in the art, they would appreciate that theremyriad reasons why a network interface reports error, includingconnection failure, modem failure, operating system failure, drop insignal-to-noise ratio detected by a modem, and etc. When a networkinterface reports an error, it is likely that it may not be able totransmit and/or receive data packets and affects established end-to-endconnection(s) going through it.

According to one of the embodiments of the present invention, as thereis only one established end-to-end connection in this embodiment, thereis no need to have an aggregated end-to-end connection to hold theestablished end-to-end connection as this reduces the size of payloadavailable while providing no significant improvement. Furthermore, asthere is only one network interface being used at one time, thecondition(s) to determine whether the first established end-to-endconnection being able to remain in the first group can also be appliedto determine whether to use a non-first network interface to transmitall data packets, regardless whether destination of data packets iscommunications router 108 or not.

For example, a first network interface of communications router 106 isdetermined to be in a first group for transmitting and receiving datapackets as it satisfies all conditions while a second network interfaceis determined to be in a second group. When the first network interfaceis about to be removed from the first group, communications router 106starts sending “warm-up” packets through the second network interface inorder to prepare the second network interface for transmitting datapackets. When the first network interface is removed from the firstgroup, the second network interface can start transmitting and receivingdata packets. When the first network interface again belongs to thefirst group, communications router 106 will use the first networkinterface to transmit and receive data packets and stop using the secondnetwork interface to transmit and receive data packets. Classificationof network interfaces into the first group and second group can beconducted in a similar manner as the classification of establishedend-to-end connections in different groups which is discussed in theembodiments described above.

When communications router 106 is not using the second network interfaceto transmit and receive data packets, communications router 106 canstill use the second network interface to transmit error correctionpackets, management packets, health-check packets and testing packets.

Network Interface without End-to-End Connection

According to one of the embodiments of the present invention andclassification is conducted on network interfaces of a communicationsdevice, instead of on end-to-end connections. The communications devicecan be a router, switch, multiplexer, host or node. This is illustratedin FIG. 11, which is viewed in conjunction with FIG. 5 for betterunderstanding, wherein FIG. 5 illustrates a block diagram of thecommunications device. Data packets are ready to be transmitted by thecommunications device in step 1101. The communications device uses afirst network interface to transmit the data packets in step 1105 whenprocessing unit 502 determines that first condition is satisfied in step1102. The communications device uses a second network interface totransmit data packets in step 1107 when processing unit 502 determinesthat second condition is satisfied in step 1103. In one variant, whenprocessing unit determines that the first condition is not satisfied instep 1102, the communications device can still use the first networkinterface to transmit data packets; when the first condition issatisfied, the communications device must use the first networkinterface to transmit data packets. Similarly, when processing unit 502determines that the second condition is not satisfied in step 1103, thecommunications device can still use the second network interface totransmit data packets; when the second condition is satisfied, thecommunications device must use the second network interface to transmitdata packets. In one variant, when processing unit 502 determines thatthe first condition is not satisfied in step 1102, the communicationsdevice stops using the first network interface to transmit data packetsin step 1104 and the same is also applied to the second condition andthe second network interface in step 1106.

The first condition and the second condition can be independent,uncorrelated, dependent, or correlated. The first condition and thesecond condition are configured by the administrator, manufacturer oruser of the communications device.

In the case that the communications device stops using the first networkinterface to transmit data packets in step 1104 if the first conditionis not satisfied, the overall network performance of the communicationsdevice will be adversely affected if the first network interface isabruptly ceased to operate. Therefore, the communications device shouldstop using first network adapter to transmit data packets while notadversely overall network performance significantly. According to one ofthe embodiments of the present invention, the communications device willmake a second network interface be ready to transmit and receive datapackets when it is about to stop using the first network interface.Therefore, the communications device makes the second network interfaceready when the first condition is about not to be satisfied. In case,the first condition is still satisfied after a predefined period of timeor the probability that the first condition not being satisfied hasreduced, the communications device does not try to keep second networkinterface active as keeping second network interface may increase powerconsumption and result in higher data usage.

The process of FIG. 11 is similar to the process of FIG. 9, and the onlydifference is that FIG. 9 illustrates a process for selectingestablished end-to-end connections for transmitting data packets, andFIG. 11 illustrates a process for selecting network interfaces fortransmitting data packets.

The conditions described earlier to classify an end-to-end connection ora network interface, including performance metric, service provider,usage metric, location, time, usage price, security, user identity,Internet Protocol address range, communication protocol, communicationtechnology and application can be used as conditions for networkinterfaces. For example, a first condition is based on a performancemetric, such that if the first network interface or the networkconnection carried by the first network interface does not satisfy thefirst condition, the communications device does not use the firstnetwork interface further. In another example, the first condition isbased on user identity that only authenticated users can use the firstnetwork interface. Therefore, the communications device does not use thefirst network interface to transmit data packets originated from anon-authenticate user.

According to one of the embodiments of the present invention, a thirdcondition is used to begin the activation process of the second networkinterface. During the activation process, the communications devicestarts activating or “warming up” the second network interface. Thereare myriad ways to activate the second network interface. For example,the second network interface can be activated by supplying electricityto it. In another example, the second network interface is activated bytransmitting packets to a communication network through it and thepackets may encapsulate data, another packet, management information,randomly generated content or predefined content. The goal foractivating the second network interface is to allow the second networkinterface to transmit data packets at the moment when the firstcondition is satisfied or when the first network interface is ceased totransmit data packets.

The third condition should be more stringent than the second condition,otherwise a scenario will happen that the third condition is satisfiedbut the second condition has not been satisfied yet that could result inthe second network interface may not be activated in time.

According to one of the embodiments of the present invention, when thecommunications device has determined that it can use the first networkinterface again to transmit data packets, the communications device willstop using the second network interface to transmit data packets andbegin using the first network interface to transmit data packets. Afourth condition is used to evaluate whether the first network interfacecan again be used to transmit data packets.

According to one of the embodiments of the present invention,conditions, including the first condition, the second condition, thethird condition, and the fourth condition, can follow the conditionsdescribed in the earlier part of the this invention.

According to one of the embodiments of the present invention, when thecommunications device transmits data through one of the networkinterfaces, the communications device can use the same network interfaceor another network interface to receive data packets.

FIG. 10 is an illustration of a preferred embodiment according to thepresent invention. At step 1001, the communications device transmitsdata packets through a first network interface. At step 1002, processingunit 502 determines whether the first network interface satisfies athird condition, which is a third predefined number of data packet dropsor loss within a third predefined time period. If the third condition issatisfied, the communications device starts activating the secondnetwork interface at step 1003. The third predefined number of datapacket drops or loss is in the range of one packet to one hundredpackets. The third predefined time period is in the range of one secondto one minute. The larger the third predefined number of data packetdrops or loss is, the more packet drops or loss is tolerated. On theother hand, the larger the third predefined time period is, the fewerpacket drops or loss is tolerated. For illustrative purpose, the thirdpredefined number of data packet loss is five packets and the thirdpredefined time period is ten seconds. Therefore, the third condition issatisfied when there are six or more packet drops within five seconds.

At step 1004, processing unit 502 determines whether a second condition,which is a second predefined number of packet drops or losses within asecond predefined period of time detected at the first networkinterface, is satisfied. If the second condition is satisfied, thecommunications device starts using the second network interface totransmit data packets at step 1005. In one variant, when thecommunications device uses the second network interface to transmit datapackets, the communications device stops using the first networkinterface to transmit data packets. Therefore the second condition canbe constructed based on the first condition that the second condition isopposite to the first condition. Such that when the first condition issatisfied, the second condition is not satisfied and when secondcondition is satisfied, the first condition is not satisfied. Forillustrative purpose only, the second predefined number of packets lossis ten packets and the second predefined time period is ten seconds atthe first network interface. Therefore, the second condition issatisfied when there have been already ten packet drops within less thanten seconds. The first condition should be configured to be fewer thanthe ten packet losses within ten seconds. As a result, when thecommunication device stops using the first network interface to transmitdata packets, the second network interface can transmit data packetssubstantially at the same time.

At step 1006, processing unit 502 determines whether the first networkinterface should be used again to transmit data packets. If a fourthcondition is satisfied, step 1007 and 1008 are performed to transmitdata packets through the first network interface again and stop usingthe second network interface to transmit data packets. The fourthcondition is a predefined number of packets loss within a fourthpredefined period of time detected at the first network interface. Forillustrative purpose only, the fourth predefined number of packets lossis five packets and the fourth predefined time period is ten seconds atthe first network interface.

In one variant, the communications device should not perform step 1006almost immediately after step 1005. This is because it is possible thatthe network connection of the first network interface has notstabilized. If step 1006 is performed when the network connection of thefirst network interface or the network interface is not stabilized, thecommunications device may toggle between using the first networkinterface and second network interface frequently. This may result insignificant adverse impact on the performance.

In one variant, the fourth conditions should not be too relaxedcomparing to the first condition. Otherwise, the communications devicemay also experience toggling between using and stopping to use the firstnetwork interface frequently. This may result in significant adverseimpact on the performance as well.

The condition(s) are stored in secondary storage 505 and are retrievedby processing unit 502 from secondary storage 505 to determine whether anetwork interface satisfies the condition(s). The first networkinterface and the second network interface are two of the plurality ofnetwork interfaces 506.

According to one of the embodiments of the present invention, processingunit 502 determines whether a network interface satisfies a condition bydetermining whether an access link, a connection, a network or anend-to-end connection that the network interface associated withsatisfies the condition. For example, if the condition is that networklatency experienced by a network interface must be less than twomilliseconds, then the condition is equivalent to network latencyobserved at an access link connected through the network interface isless than two milliseconds.

Alert Message

According to one of the embodiments of the present invention, when anestablished end-to-end connection leaves or joins a group, or when anestablished end-to-end connection stops satisfying a condition, at leastone alert message is generated by communications router 106 and sent toa predefined destination. An alert message can be an email, an instantmessage, a short message service (SMS), a phone call, a message shown ina web page, a popup message at a web page, an alarm, a sound, a blinkinglight, a light-emitting diode (LED) being turned on and other indicatorsthat can be used to indicating an event has occurred. A predefineddestination can be an email address, an IP address, a LED, a speaker, ascreen, a console, a network node, a host, a mobile phone, a laptop andany electronic device that can receive the alert message. For example,when a first established end-to-end connection at communications router106 has left the first group, a SMS is sent to the administrator ofcommunications router 106. In another example, when a first establishedend-to-end connection at communications router 106 has left the firstgroup, at least two alert messages are generated by communicationsrouter 106. The at least two alert messages include a pop-up messagegenerated on a webpage and an instant message sent to the user and/oradministrator of communications router 106. In another example, when thefirst established end-to-end connection at communications router 106 hasrejoined the first group, a LED light on communications router 106 willblink for a few second to indicate that the first established end-to-endconnection has rejoined the first group.

Input of Conditions

The condition(s) can be entered into communications router 106 through aconsole, a connection and/or a removable computer readable storagemedium. In particular, a console is a serial console that a laptop, forexample, is connected to communications router 106 through the serialconsole to enter, configure, manage and remove condition(s). Aconnection, in particular, is connected to communications router 106through an Ethernet cable. One end of the Ethernet cable is connected tocommunications router 106 and the other end of the Ethernet cable isconnected to a network node, which provides connection to othernetworks, including the Internet. Thus, for example, an administrator ofcommunications router 106 is able to log on to communications router 106to enter, configure, manage and remove conditions. Alternatively, theconnection is a wireless connection implementing a wirelesscommunication standard including LTE, WI-FI, HighSpeed Downlink PacketAccess (HSDPA) communication standard, Zigbee, UWB, 3G, WiMAX, Bluetoothand GPRS. A removable computer readable storage medium includes a Flashcard, a subscriber identity module (SIM) card, a secure digital (SD)card or any removable computer readable storage medium that allowsinstructions being uploaded to communications router 106.

According to one of the embodiments of the present invention, when anadministrator accesses communications router 106 through a connection,the administrator is presented with a web interface, command lineinterface and/or application programming interface (API). Thereforeconditions can be inputted remotely or locally through the webinterface, command line interface and/or API and can be stored insecondary storage 505.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A illustrates a network environment according to one of theembodiments of the present invention.

FIG. 1B illustrates a system adapted according to embodiments configuredto optimize the throughput of bonded multiple variable bandwidthconnections by adjusting a tunnel bandwidth weighting schema during adata transfer session.

FIG. 2A shows a high level flow diagram of operation of a systemdepicting a method for increasing throughput of a bonded connection.

FIG. 2B illustrates an embodiment where, after a weighting method isimplemented, the packets are distributed based, at least in part, on themodified weight of the tunnels.

FIG. 3 illustrates the type of information which may be encapsulated ina transmitted IP packet according to one of the embodiments.

FIG. 4 is a tree-diagram illustrating how established end-to-endconnections are classified into groups according to one of theembodiments.

FIG. 5 is an illustrative block diagram of a communications routeraccording to one of the embodiments.

FIG. 6A is a flowchart illustrating a process used to classify anestablished end-to-end connection into one of the groups according toone of the embodiments.

FIG. 6B illustrates a process of classifying established end-to-endconnections into multiple groups based on multiple conditions accordingto one of the embodiments.

FIG. 6C illustrates a process of classifying established end-to-endconnections into multiple groups based on multiple conditions accordingto one of the embodiments, wherein the multiple conditions are all basedon the same metric.

FIG. 7 illustrates rankings of multiple groups of established end-to-endconnections at a communications router according to one of theembodiments.

FIG. 8 is a flowchart illustrating a process according to one of theembodiments.

FIG. 9 is a flowchart illustrating a process of selecting establishedend-to-end connection(s) for transmitting data packets according to oneof the embodiments.

FIG. 10 is an illustration of a preferred embodiment according to thepresent invention.

FIG. 11 is a flowchart illustrating a process of selecting networkinterface(s) for transmitting data packets according to one of theembodiments.

1. A method for transmitting and receiving data packets by a wirelessnetwork node to and from a network node through a plurality of wirelessnetwork interfaces, comprising: a) establishing a first aggregatedend-to-end connection using a first group of wireless networkinterfaces; wherein the first aggregated end-to-end connection isestablished through at least two of the plurality of wireless networkinterfaces; b) transmitting and receiving data packets through the firstgroup of wireless network interfaces; c) when a first condition is notsatisfied: i) allowing a second group of network interface(s) totransmit and receive data packets; ii) when a predefined number of datapacket drops for transmitting the data packets at the first aggregatedend-to-end connection within a predefined time period is detected,transmitting and receiving data packets through the second group ofwireless network interface(s); wherein the data packet drops fortransmitting the data packets is monitored continuously; and wherein thefirst condition is based on one or more of the following: performancemetric, service provider, location, time, usage price, security, useridentity, Internet Protocol (IP) address range, communication protocol,communication technology, application, and device.
 2. The method ofclaim 1, further comprising: establishing a second aggregated end-to-endconnection using the second group of wireless network interface(s) afterstep c-i; wherein the data packets are transmitted through the secondaggregated end-to-end connection at step c-ii.
 3. The method of claim 1,further comprising starting to use the first group of wireless networkinterfaces again when a second condition is satisfied.
 4. The method ofclaim 3, further comprising stopping to use the second group of wirelessnetwork interface(s) after the second condition has been satisfied for aperiod of time.
 5. The method of claim 1, further comprising: sendingwarm-up packets through the second group of wireless networkinterface(s) at step c-i.
 6. The method of claim 1, further comprising:sending an alert message when the first aggregated end-to-end connectionstops satisfying the first condition.
 7. The method of claim 1, furthercomprising: stopping to transmit data packets, which belong to aTransmission Control Protocol (TCP) session, through the firstaggregated end-to-end connection when the first aggregated end-to-endconnection stops satisfying the first condition.
 8. The method of claim1, further comprising: allowing transmission of non-TCP data packetsthrough the first aggregated end-to-end connection when the firstaggregated end-to-end connection stops satisfying the first condition.9. The method of claim 1, further comprising: transmitting errorcorrection packets through a wireless network interface, wherein thewireless network interface is one of the plurality of wirelessinterfaces.
 10. A method for transmitting and receiving data packets bya wireless network node to a network node through a plurality ofwireless network interfaces, comprising: a) establishing a firstaggregated end-to-end connection using a first group of wireless networkinterfaces; wherein the first aggregated end-to-end connection isestablished through at least two of the plurality of wireless networkinterfaces; b) transmitting and receiving data packets through a firstgroup of wireless network interfaces; c) when a first condition is notsatisfied: i) allowing a second group of network interface(s) totransmit and receive data packets; ii) when a predefined number of datapacket drops for transmitting the data packets at the first aggregatedend-to-end connection within a predefined time period is detected; iii)transmitting and receiving data packets through the second group ofwireless network interface(s); iv) when a second condition is satisfied,starting to use the first group of wireless network interfaces again v)when the second condition has been satisfied for a period of time,stopping to use the second group of wireless network interfaces; whereinthe data packet drops for transmitting the data packets is monitoredcontinuously; and wherein the first condition is selected from a groupconsisting of performance metric, service provider, location, time,usage price, security, user identity, Internet Protocol (IP) addressrange, communication protocol, communication technology, application,and device.
 11. The method of claim 10, further comprising: sendingwarm-up packets through the second group of wireless networkinterface(s) at step c-i.
 12. A wireless network node for transmittingand receiving data packets to and from a network node through aplurality of network interfaces, the wireless network node comprising: aplurality of network interfaces; at least one processing unit; at leastone non-transitory computer readable storage medium storing programinstructions executable by the at least one processing unit for: a)establishing a first aggregated end-to-end connection using a firstgroup of wireless network interfaces; wherein the first aggregatedend-to-end connection is established through at least two of theplurality of wireless network interfaces; d) transmitting and receivingdata packets through the first group of wireless network interfaces; e)when a first condition is not satisfied: i) allowing a second group ofnetwork interface(s) to transmit and receive data packets; ii) when apredefined number of data packet drops for transmitting the data packetsat the first aggregated end-to-end connection within a predefined timeperiod is detected, transmitting and receiving data packets through thesecond group of wireless network interface(s); wherein the data packetdrops for transmitting the data packets is monitored continuously; andwherein the first condition is based on one or more of the following:performance metric, service provider, location, time, usage price,security, user identity, Internet Protocol (IP) address range,communication protocol, communication technology, application, anddevice.
 13. The wireless network node of claim 11, wherein the at leastone non-transitory computer readable storage medium further storingprogram instructions executable by the at least one processing unit forestablishing a second aggregated end-to-end connection using the secondgroup of wireless network interface(s) after step c-i; wherein the datapackets are transmitted through the second aggregated end-to-endconnection at step c-ii.
 14. The wireless network node of claim 11,wherein the at least one non-transitory computer readable storage mediumfurther storing program instructions executable by the at least oneprocessing unit for starting to use the first group of wireless networkinterfaces again when a second condition is satisfied.
 15. The wirelessnetwork node of claim 11, wherein the at least one non-transitorycomputer readable storage medium further storing program instructionsexecutable by the at least one processing unit for stopping to use thesecond group of wireless network interface(s) after the second conditionhas been satisfied for a period of time.
 16. The wireless network nodeof claim 11, wherein the at least one non-transitory computer readablestorage medium further storing program instructions executable by the atleast one processing unit for sending warm-up packets through the secondgroup of wireless network interface(s) at step c-i.
 17. The wirelessnetwork node of claim 11, wherein the at least one non-transitorycomputer readable storage medium further storing program instructionsexecutable by the at least one processing unit for sending an alertmessage when the first aggregated end-to-end connection stops satisfyingthe first condition.
 18. The wireless network node of claim 11, whereinthe at least one non-transitory computer readable storage medium furtherstoring program instructions executable by the at least one processingunit for stopping to transmit data packets, which belong to aTransmission Control Protocol (TCP) session, through the firstaggregated end-to-end connection when the first aggregated end-to-endconnection stops satisfying the first condition.
 19. The wirelessnetwork node of claim 11, wherein the at least one non-transitorycomputer readable storage medium further storing program instructionsexecutable by the at least one processing unit for allowing transmissionof non-TCP data packets through the first aggregated end-to-endconnection when the first aggregated end-to-end connection stopssatisfying the first condition.
 20. The wireless network node of claim11, wherein the at least one non-transitory computer readable storagemedium further storing program instructions executable by the at leastone processing unit for transmitting error correction packets through awireless network interface, wherein the wireless network interface isone of the plurality of wireless interfaces.